feat: add CONTRIBUTING.md and structural PR validation CI#62
Open
sendtoshailesh wants to merge 1 commit intomainfrom
Open
feat: add CONTRIBUTING.md and structural PR validation CI#62sendtoshailesh wants to merge 1 commit intomainfrom
sendtoshailesh wants to merge 1 commit intomainfrom
Conversation
Implements scope of #17 (de-scoped): - CONTRIBUTING.md: skill contribution guide, SKILL.md schema, PR process, Code of Conduct link - .github/workflows/pr-validation.yml: structural validation + markdownlint on every PR - scripts/validate-structure.js: frontmatter, naming, cross-reference, link, and section checks - .markdownlint.json: shared lint config The eval framework (originally section 3 of #17) is split out to #61 and is not part of this PR. Supersedes #40.
arnaudlh
force-pushed
the
feat/contributing-and-pr-validation
branch
from
4e3b42e to
0dec1ad
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds contributor-facing documentation plus automated PR checks to validate the repo’s skill/agent Markdown structure and enforce Markdown linting, aligning with issue #17’s reduced scope.
Changes:
- Added
CONTRIBUTING.mddescribing skill contribution structure, frontmatter schema, and PR process/CI checks. - Added a PR workflow to run structural validation and markdownlint on pull requests to
main. - Implemented
scripts/validate-structure.jsto validate skill/agent frontmatter, required sections, cross-references, and relative links.
Show a summary per file
| File | Description |
|---|---|
scripts/validate-structure.js |
New Node.js validator for skills/agents structure, frontmatter, sections, cross-references, and relative links. |
CONTRIBUTING.md |
New contributor guide describing how to add skills and what CI validates. |
.markdownlint.json |
Shared markdownlint configuration for repo-wide Markdown linting. |
.github/workflows/pr-validation.yml |
New PR workflow running structural validation + markdownlint on every PR to main. |
Copilot's findings
Comments suppressed due to low confidence (1)
scripts/validate-structure.js:257
- In
checkCrossReferences(), slash-commands extracted from agent content are never validated or even warned on (the loop body is empty). This is dead code and also contradicts the intended “cross-reference integrity” check for agent docs. Either remove the unused extraction or implement the same warn/error behavior used for skills.
// Check slash-command references in content
const slashCommands = extractSlashCommands(content);
for (const cmd of slashCommands) {
if (!skillNames.has(cmd)) {
// Only warn — some slash commands may reference non-skill entities
// or be examples in documentation
}
}
- Files reviewed: 4/4 changed files
- Comments generated: 4
Comment on lines
+74
to
+82
| // Match /skill-name patterns that look like intentional skill invocations | ||
| // Exclude common false positives: file paths, URLs, API paths | ||
| const PATH_PREFIXES = new Set([ | ||
| 'etc', 'dev', 'usr', 'var', 'tmp', 'home', 'opt', 'bin', 'sbin', | ||
| 'api', 'v1', 'v2', 'v3', 'subscriptions', 'providers', 'admin', | ||
| ]); | ||
| const matches = content.match(/(?:^|\s)\/([a-z][a-z0-9-]*)/gm) || []; | ||
| return matches | ||
| .map((m) => m.trim().slice(1)) |
Comment on lines
+124
to
+150
| function checkSkillFrontmatter(skillDirs) { | ||
| console.log('\n🏷️ Skill frontmatter validation:'); | ||
| for (const dir of skillDirs) { | ||
| const skillMd = path.join(SKILLS_DIR, dir, 'SKILL.md'); | ||
| if (!fs.existsSync(skillMd)) continue; | ||
|
|
||
| const parsed = parseFrontmatter(skillMd); | ||
| if (!parsed) { | ||
| error(`${dir}/SKILL.md: Could not parse YAML frontmatter`); | ||
| continue; | ||
| } | ||
|
|
||
| const { data: fm } = parsed; | ||
|
|
||
| if (!fm.name) { | ||
| error(`${dir}/SKILL.md: Missing required frontmatter field 'name'`); | ||
| } else if (fm.name !== dir) { | ||
| error(`${dir}/SKILL.md: Frontmatter 'name' is '${fm.name}' but directory is '${dir}'`); | ||
| } | ||
|
|
||
| if (!fm.description) { | ||
| error(`${dir}/SKILL.md: Missing required frontmatter field 'description'`); | ||
| } | ||
| } | ||
| if (errors.length === 0) { | ||
| ok('All skills have valid frontmatter with name and description'); | ||
| } |
Comment on lines
+105
to
+114
| 5. **CI checks run automatically** — The PR validation workflow verifies: | ||
| - YAML frontmatter has required fields (`name`, `description` for skills; `description` for agents) | ||
| - Skill `name` matches its parent directory name | ||
| - All skill/agent directories use kebab-case | ||
| - Every skill directory contains a `SKILL.md` file | ||
| - Skills have `## When to Use` and `## Procedure` sections | ||
| - Agents have a `## Warning` disclaimer section | ||
| - Cross-references (slash-commands `/skill-name`) map to existing skill directories | ||
| - Relative markdown links resolve to real file paths | ||
| - Markdown passes linting (markdownlint) |
| node-version: '20' | ||
|
|
||
| - name: Install markdownlint-cli | ||
| run: npm install -g markdownlint-cli |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #17.
Adds a contributor guide and a structural PR validation pipeline so community skill contributions can be verified automatically.
This is a re-scoped replacement for #40, which mixed three concerns. The exploratory eval framework portion has been split into #61 and intentionally excluded from this PR.
Changes
CONTRIBUTING.md— skill contribution guide (directory structure,SKILL.mdfrontmatter schema, naming conventions), maintainer-curated agent change process, PR process, and a link to the Microsoft Open Source Code of Conduct..github/workflows/pr-validation.yml— active workflow on every PR tomain:structure-checkjob runsscripts/validate-structure.js.markdownlintjob runs across*.md(excludingwebsite/node_modules,website/build,website/docs).scripts/validate-structure.js— checks frontmatter required fields, name-directory consistency, kebab-case directory naming,SKILL.mdpresence, required sections (## When to Use,## Procedure;## Warningfor agents), agent/skill cross-reference integrity, and relative-link resolution..markdownlint.json— shared lint config.All checks pass against the current repo state.
What is not in this PR
The Agent/Skill eval framework (originally section 3 of #17) is tracked separately in #61 and is intentionally out of scope here. The earlier spike under
evals/from #40 is not carried over; #61 will revisit it from scratch.Acceptance criteria mapping (#17, reduced scope)
CONTRIBUTING.mdexists with clear skill contribution guidelines andSKILL.mdschema..yml, not.example.yml).Notes for reviewers
CONTRIBUTING.md).Related